{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 基于机器学习的文本分类–朴素贝叶斯\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from tqdm import tqdm\n",
    "import time\n",
    "from numpy import *\n",
    "import pandas as pd"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "直接通过python自带的open()函数读取文件，并建立对应词典，设定停用词，这里的停用词选择了words字典中出现在100000个文档以上的所有词。训练集取前19万个文档，测试集取最后一万个文档。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "train_df = open('E:\\\\code\\\\Test\\\\data\\\\train_set.csv').readlines()[1:]\n",
    "train = train_df[0:190000]\n",
    "test = train_df[190000:200000]\n",
    "true_test = open('E:\\\\code\\\\Test\\\\data\\\\train_set.csv').readlines()[1:]\n",
    "tag = {str(i):0 for i in range(0,14)}\n",
    "sen = {str(i):{} for i in range(0,14)}\n",
    "words={}\n",
    "stop_words =set(stopwords.words('english'))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "接着，需要建立标签词典和句子词典，用tqdm函数来显示进度。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 200000/200000 [01:55<00:00, 1732.93it/s]\n"
     ]
    }
   ],
   "source": [
    "for line in tqdm(train_df):\n",
    "    cur_line = line.split('\\t')\n",
    "    cur_tag = cur_line[0]\n",
    "    tag[cur_tag] += 1\n",
    "    cur_line = cur_line[1][:-1].split(' ')\n",
    "    for i in cur_line:\n",
    "        if i not in words:\n",
    "            words[i] = 1\n",
    "        else:\n",
    "            words[i] += 1\n",
    "        if i not in sen[cur_tag]:\n",
    "            sen[cur_tag][i] = 1\n",
    "        else:\n",
    "            sen[cur_tag][i] += 1"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "为了便于计算，定义了如下函数，其中mul()用来计算列表中所有数的乘积，prob_clas() 用来计算P ( C i ∣ D o c ) ，用probability(）来计算P ( t i ∣ C i ) ，在probability() 函数中，将输出结果中分子+1，分母加上字典长度，实现拉普拉斯平滑处理。\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "outputs": [],
   "source": [
    "def mul(l):\n",
    "    res = 1\n",
    "    for i in l:\n",
    "        res *= i\n",
    "    return res\n",
    "def prob_clas(clas):\n",
    "    return tag[clas]/(sum([tag[i] for i in tag]))\n",
    "def probability(char,clas):  #P(特征|类别)\n",
    "    if char not in sen[clas]:\n",
    "        num_char = 0\n",
    "    else:\n",
    "        num_char = sen[clas][char]\n",
    "    return (1+num_char)/(len(sen[clas])+len(words))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "在做好所有准备工作，定义好函数后，分别对测试集中的每一句话计算十四个标签对应概率，并将概率最大的标签储存在预测列表中，用tqdm函数来显示进度。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 200000/200000 [1:16:31<00:00, 43.56it/s]\n"
     ]
    }
   ],
   "source": [
    "PRED = []\n",
    "for line in tqdm(true_test):\n",
    "    result = {str(i):0 for i in range(0,14)}\n",
    "    cur_line = line[:-1].split(' ')\n",
    "    clas = cur_tag\n",
    "    for i in result:\n",
    "        prob = []\n",
    "        for j in cur_line:\n",
    "            if j in stop_words:\n",
    "                continue\n",
    "            prob.append(log(probability(j,i)))\n",
    "        result[i] = log(prob_clas(i))+sum(prob)\n",
    "    for key,value in result.items():\n",
    "        if(value == max(result.values())):\n",
    "            pred = int(key)\n",
    "    PRED.append(pred)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "最后把结果储存在csv文件中"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "outputs": [],
   "source": [
    "res=pd.DataFrame()\n",
    "res['label']=PRED\n",
    "res.to_csv('test_TL.csv')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}